Much ado about something: a response to "COVID-19: underpowered randomised trials, or no randomised trials?"

Non-pharmaceutical interventions (NPI) for infectious diseases such as COVID-19 are particularly challenging given the complexities of what is both practical and ethical to randomize. We are often faced with the difficult decision between having weak trials or not having a trial at all. In a recent article, Dr. Atle Fretheim argues that statistically underpowered studies are still valuable, particularly in conjunction with other similar studies in meta-analysis in the context of the DANMASK-19 trial, asking "Surely, some trial evidence must be better than no trial evidence?" However, informative trials are not always feasible, and feasible trials are not always informative. In some cases, even a well-conducted but weakly designed and/or underpowered trial such as DANMASK-19 may be uninformative or worse, both individually and in a body of literature. Meta-analysis, for example, can only resolve issues of statistical power if there is a reasonable expectation of compatible well-designed trials. Uninformative designs may also invite misinformation. Here, we make the case that-when considering informativeness, ethics, and opportunity costs in addition to statistical power-"nothing" is often the better choice.

Association between convalescent plasma treatment and mortality in COVID-19: a collaborative systematic review and meta-analysis of randomized clinical trials.

BACKGROUND: Convalescent plasma has been widely used to treat COVID-19 and is under investigation in numerous randomized clinical trials, but results are publicly available only for a small number of trials. The objective of this study was to assess the benefits of convalescent plasma treatment compared to placebo or no treatment and all-cause mortality in patients with COVID-19, using data from all available randomized clinical trials, including unpublished and ongoing trials (Open Science Framework, https://doi.org/10.17605/OSF.IO/GEHFX ). METHODS: In this collaborative systematic review and meta-analysis, clinical trial registries (ClinicalTrials.gov, WHO International Clinical Trials Registry Platform), the Cochrane COVID-19 register, the LOVE database, and PubMed were searched until April 8, 2021. Investigators of trials registered by March 1, 2021, without published results were contacted via email. Eligible were ongoing, discontinued and completed randomized clinical trials that compared convalescent plasma with placebo or no treatment in COVID-19 patients, regardless of setting or treatment schedule. Aggregated mortality data were extracted from publications or provided by investigators of unpublished trials and combined using the Hartung-Knapp-Sidik-Jonkman random effects model. We investigated the contribution of unpublished trials to the overall evidence. RESULTS: A total of 16,477 patients were included in 33 trials (20 unpublished with 3190 patients, 13 published with 13,287 patients). 32 trials enrolled only hospitalized patients (including 3 with only intensive care unit patients). Risk of bias was low for 29/33 trials. Of 8495 patients who received convalescent plasma, 1997 died (23%), and of 7982 control patients, 1952 died (24%). The combined risk ratio for all-cause mortality was 0.97 (95% confidence interval: 0.92; 1.02) with between-study heterogeneity not beyond chance (I2 = 0%). The RECOVERY trial had 69.8% and the unpublished evidence 25.3% of the weight in the meta-analysis. CONCLUSIONS: Convalescent plasma treatment of patients with COVID-19 did not reduce all-cause mortality. These results provide strong evidence that convalescent plasma treatment for patients with COVID-19 should not be used outside of randomized trials. Evidence synthesis from collaborations among trial investigators can inform both evidence generation and evidence application in patient care.

Impact Evaluation of Coronavirus Disease 2019 Policy: A Guide to Common Design Issues.

Policy responses to coronavirus disease 2019 (COVID-19), particularly those related to nonpharmaceutical interventions, are unprecedented in scale and scope. However, evaluations of policy impacts require a complex combination of circumstance, study design, data, statistics, and analysis. Beyond the issues that are faced for any policy, evaluation of COVID-19 policies is complicated by additional challenges related to infectious disease dynamics and a multiplicity of interventions. The methods needed for policy-level impact evaluation are not often used or taught in epidemiology, and they differ in important ways that may not be obvious. Methodological complications of policy evaluations can make it difficult for decision-makers and researchers to synthesize and evaluate the strength of the evidence in COVID-19 health policy papers. Here we 1) introduce the basic suite of policy-impact evaluation designs for observational data, including cross-sectional analyses, pre-/post- analyses, interrupted time-series analysis, and difference-in-differences analysis; 2) demonstrate key ways in which the requirements and assumptions underlying these designs are often violated in the context of COVID-19; and 3) provide decision-makers and reviewers with a conceptual and graphical guide to identifying these key violations. Our overall goal is to help epidemiologists, policy-makers, journal editors, journalists, researchers, and other research consumers understand and weigh the strengths and limitations of evidence.

Association of Convalescent Plasma Treatment With Clinical Outcomes in Patients With COVID-19: A Systematic Review and Meta-analysis.

Importance: Convalescent plasma is a proposed treatment for COVID-19. Objective: To assess clinical outcomes with convalescent plasma treatment vs placebo or standard of care in peer-reviewed and preprint publications or press releases of randomized clinical trials (RCTs). Data Sources: PubMed, the Cochrane COVID-19 trial registry, and the Living Overview of Evidence platform were searched until January 29, 2021. Study Selection: The RCTs selected compared any type of convalescent plasma vs placebo or standard of care for patients with confirmed or suspected COVID-19 in any treatment setting. Data Extraction and Synthesis: Two reviewers independently extracted data on relevant clinical outcomes, trial characteristics, and patient characteristics and used the Cochrane Risk of Bias Assessment Tool. The primary analysis included peer-reviewed publications of RCTs only, whereas the secondary analysis included all publicly available RCT data (peer-reviewed publications, preprints, and press releases). Inverse variance-weighted meta-analyses were conducted to summarize the treatment effects. The certainty of the evidence was assessed using the Grading of Recommendations Assessment, Development, and Evaluation. Main Outcomes and Measures: All-cause mortality, length of hospital stay, clinical improvement, clinical deterioration, mechanical ventilation use, and serious adverse events. Results: A total of 1060 patients from 4 peer-reviewed RCTs and 10 722 patients from 6 other publicly available RCTs were included. The summary risk ratio (RR) for all-cause mortality with convalescent plasma in the 4 peer-reviewed RCTs was 0.93 (95% CI, 0.63 to 1.38), the absolute risk difference was -1.21% (95% CI, -5.29% to 2.88%), and there was low certainty of the evidence due to imprecision. Across all 10 RCTs, the summary RR was 1.02 (95% CI, 0.92 to 1.12) and there was moderate certainty of the evidence due to inclusion of unpublished data. Among the peer-reviewed RCTs, the summary hazard ratio was 1.17 (95% CI, 0.07 to 20.34) for length of hospital stay, the summary RR was 0.76 (95% CI, 0.20 to 2.87) for mechanical ventilation use (the absolute risk difference for mechanical ventilation use was -2.56% [95% CI, -13.16% to 8.05%]), and there was low certainty of the evidence due to imprecision for both outcomes. Limited data on clinical improvement, clinical deterioration, and serious adverse events showed no significant differences. Conclusions and Relevance: Treatment with convalescent plasma compared with placebo or standard of care was not significantly associated with a decrease in all-cause mortality or with any benefit for other clinical outcomes. The certainty of the evidence was low to moderate for all-cause mortality and low for other outcomes.